Recently, various technologies have emerged that allow users to experience a blend of reality and virtual worlds along an augmented reality continuum. For example, head-mounted display (HMD) devices may include various sensors that allow the HMD device to display a blend of reality and virtual objects on the HMD device as augmented reality, or block out the real world view to display only virtual reality. Whether for virtual or augmented reality, is often desired to present images to multiple users at the same time in order to provide an integrated virtual or augmented reality experience. However, a lack of unification of the images when displaying virtual or augmented reality images to multiple users simultaneously can make the images appear less realistic, which negatively impacts the interactive experience for the users.